Information theoretical estimators toolbox

Since the pioneering work of Shannon, entropy, mutual information, association, divergence measures and kernels on distributions have found a broad range of applications in many areas of machine learning. Entropies provide a natural notion to quantify the uncertainty of random variables, mutual information and association indices measure the dependence among its arguments, divergences and kernels offer efficient tools to define the ‘distance’ and the inner product of probability measures, respectively.